Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding

AbstractThere has been an increasing interest in multi-task learning for video understanding recent years. In this work, we propose a generalized notion of by incorporating both auxiliary tasks that the model should perform well on and adversarial not on. We employ Necessary Condition Analysis (NCA) as data-driven approach deciding what category these fall in. Our novel proposed framework, Adversarial Multi-Task Neural Networks (AMT), penalizes tasks, determined NCA to be scene recognition Holistic Video Understanding (HVU) dataset, improve action recognition. This upends common assumption always encouraged do all learning. Simultaneously, AMT still retains benefits generalization existing methods uses object task aid introduce two challenging Scene-Invariant test splits HVU, where is evaluated action-scene co-occurrences encountered training. show our improves accuracy \(\sim \)3% encourages attend features instead correlation-biasing features. KeywordsVideo understandingActivity recognitionInvariant feature learningMulti-task